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Description 

FIELD OF THE INVENTION 

The present invention is related to a data process- 
ing system based on the concept of multilayer neural 
network. 

BACKGROUND OF THE INVENTION 

The neural network in such kind of data processing 
systems is constructed in layer state by settling parallely 
neural cell model 1 in fig. 3 ("neuron", hereinafter). Every 
neuron in each layer is connected by synapses with all 
the neurons in adjacent other layer and inputs and out- 
puts data. Concerning to neuron 1, weights W1, W2, 
W3, ...,Wnaremultipliedtodatal1,l2, 13, .... In inputted 
from outside, and data O is outputted corresponding to 
the comparison between the sum of the multiplication 
and threshold 9. 

Various methods are possible for carrying out the 
comparison. When normalized function l[f] is adopted, 
output data O is expressed as activation pattern defined 
in formula (1). 

0=1 [SWn-ln-6] (1) 
When EWn-ln is more than or equal to threshold 6, the 
neuron ignites and output data O is "1 ": when XWn-ln is 
less than threshold 6, output data O is "0". 

Conventional neural networks form neural layers by 
settling parallely such neurons and by connecting the 
neural layers in series. A neural network is comprised 
of, for example, 3 layers: input layer, middle layer and 
output layer, as Perceptrons suggested by Rosenblatt: 
every neuron in each layer is connected with all neurons 
in adjacent other layers by synapses. 

Document IEEE INTERNATIONAL CONFERENCE 
ON NEURAL NETWORKS, vol. 1, 24 July 1988, SAN 
DIEGO, US, pages 465-472, SHEPANSKI: 'Fast learn- 
ing artificial neural systems: Multilayer Perception train- 
ing using optimal estimation' describes loading informa- 
tion in a multilayer perception using methods of optimal 
estimation theory. Training occurs as a simultaneous 
calculation where an optimal set of interconnection 
weights, linking two different layers, are determined us- 
ing a least squares criterion, employing standard pseu- 
doin verse matrix techniques. 

In document IEEE FIRST INTERNATIONAL CON- 
FERENCE ON NEURAL NETWORKS, vol. 4, 21 June 
1987, SAN DIEGO, USA, pages 635-642, MC AULAY 
'Engeneering design neural networks using split inver- 
sion learning' a learning algorithm is proposed that op- 
timizes weights for all layers simultaneously in a global 
manner. Said algorithm can be computed for each layer 
in parallel. 
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SUMMARY OF THE INVENTION 

In such a data processing system, the operation of 
adapting the weights corresponding to the connections 

s between neurons is called "learning". It is an important 
problem to guarantee its realization and to perform it ef- 
ficiently. For example, in the back-propagation method, 
which attracted attention in these years, there are prob- 
lems of getting out of local minimum and of convergence 

10 time. Especially the tendency is strong in neural net- 
works including a lot of middle layers. 

The present invention intends to solve the above 
problems of the prior art and has the object of providing 
a learning method with the ability of efficiently executing 

is learning of the middle layer. 

A learning method according to the present inven- 
tion works as set out in claim 1 . 

In the case that a lot of middle layers are settled, 
the number of middle layers is increased sequentially 

20 by adding a new middle layer when the learning in a mid- 
dle layer is concluded on a predetermined step and ex- 
ecuting the same learning to said new middle layer. 
Here, the number of neurons contained in the input and 
output layer is decided by data capacity of the network 

25 the number of data is decided by the number of neurons 
contained in the whole neural network. Therefore, it is 
indispensable to increase the number of middle layers 
to in order to be able of processing a lot of data: 

30 BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 shows the approximate structure of character 
recognition system applied the first embodiment of the 
present invention. 
35 Figures 2A and 2B show the outline structure of 
neuron in input layer, middle layer and output layer. 

Fig. 3 shows the outline structure of an embodiment 
of a neuron. 

Fig. 4 shows the diagram of the relationship be- 
40 tween the number of learning and weight. 

1 andN...neuron, 31. ..input layer, 32.. middle layer, 
33.. .output layer. 

PREFERRED EMBODIMENT OF THE PRESENT 
45 INVENTION 

Hereinafter, the present invention is explained with 
referring to the attached drawings. 

Fig. 1 shows a character recognition system corn- 
so prising a data processing system related to the first em- 
bodiment. The character recognition system comprises 
video camera 10, pre-processing system 20, data 
processing system 30, post-processing system 40 and 
display 50. Video camera 10 is settled to input charac- 
55 ters/letters and connected to pre-processing system 20. 
Pre-processing system 20,is, for example, well-known 
image processing system, which extracts the character- 
istics of inputted character/letter (the number of edge 
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points or branch points, for instance). and output the data 
to data processing system 30. Data processing system 
30, constructs neural network as described later, recog- 
nizes a character/letter based on the characteristics da- 
ta inputted from pre-processing system 20, and outputs 
the data according to the recognition result to post- 
processing system 40. Recognition signal is character 
codes for instance. Post-processing system 40 stores 
the output data as, for example, word processor data 
and simultaneously outputs to display 50. Display 50 is 
constructed from CRT and indicates the characters/let- 
ters recognized by data processing system 40 on the 
display. 

The neural network to construct data processing 
system 30 is constructed as a part of hardware of a com- 
puter. Data processing system 30 is expressed in Fig. 
1 as a model. It comprises input layer 31 , middle layer 
32, and output layer 33: middle layer 32 is settle be- 
tween input layer 31 and output layer 33. In the present 
embodiment, each layer 31 , 32, and 33 comprises many 
number of neurons of N: neuron "N"s in input layer 31 
are connected to all the neurons of N in middle layer 32, 
and those in middle layer 32 are connected to all the 
neurons of N in output layer 33. 

As mentioned above referring to Fig. 3, each neuron 
N outputs the data of "1 " or "0" according to the normal- 
ized function in formula (1 ). Neuron N is constructed by 
means of an amplifier, for example. Weight Wn to be 
multiplied to the data to be inputted to each neuron N is 
obtained by changeable register connected to input ter- 
minal. Threshold function is realized by switching ele- 
ment. That is, learning is executed amending output da- 
ta and changing weight Wn by changing changeable 
register according to output data of each neuron. 

Figs. 2A and 2B (a), (b), (c) show input layer 31 , 
middle layer 32 and output layer 33 as a model: among 
the three, input layer 31 possesses the most neurons 
and output layer 33 possesses the least. Here, for a sim- 
ple explanation, it is supposed that the number of neu- 
rons in input layer 31 is 64, in middle layer 32 is 49, and 
in output layer 33 is 36: neurons are arranged 8, 7 and 
6 in lateral and longitudinal directions in input layer 31, 
middle layer 32, and output layer 33, respectively. Here, 
it is presumed that the point in the left below the edge 
is the origin, and that the neuron in the location of i-th 
left and j-th upper is Nij. 

Each neuron in input layer 31 ignites by character- 
istics data obtained through video camera 1 0. When ig- 
nition combination of N11, N12, N21 and N22 expresses 
the number of edge points, and N1 3, N14, N23 and N24 
expresses the number of branch point, the ignition pat- 
tern of neurons in input layer 31 is decided artificially 
according to characters/letters to be inputted. 

On the other hand in output layer 33, a character/ 
letter is expressed by neuron Nii on diagonal line from 
the edge on lower left to the edge on upper right (neu- 
rons surrounded by broken line "D"). That is, the neu- 
rons on the diagonal line express the character code of 



the character/letter, and the ignition pattern of neuron in 
output layer 33 is decided artificially. In the present em- 
bodiment, the kind of ignition pattern is 64 for neuron Nii 
on diagonal line. Therefore, it is possible to recognize 

5 64 kinds of characters/letters: recognition of alphabets 
is possible, for example. 

Before executing learning of character/letter recog- 
nition, neurons in output layer 33 do not ignite even if 
character/letter data is inputted to data processing sys- 

10 tern 30. This neuron can ignite by learning: its learning 
is concluded when a predetermined ignition pattern ap- 
pears according to inputted character/letter data. Input 
and output pattern for learning is the representative data 
to be processed in neural network: the data to be actu^ 

is ally processed exists widely. Learning has been execut- 
ed until association for the representative input and out- 
put data is performed adequately. When learning is con- 
cluded, input layer 31 and output layer 33 show ignition 
pattern decided artificially according to inputted charac- 

20 ter/letter data as mentioned above. It is presumed that 
ignition pattern in each layer of input layer 31 , middle 
layer 32 and output layer 33 changes smoothly in these 
layers of 31 , 32 and 33. In the present embodiment, the 
weights in middle layer 32 and output layer 33 are 

25 changed in order for the ignition pattern in these layers 
to change smoothly on learning process. 

First, it is decided if neuron Nij in middle layer 32 
ignites or not considering the ignition of plural neurons 
in input layer 31 and output layer 33 corresponding to , 

30 each other. That is, a) when neurons ignite or are to be 
ignited with more than a predetermined rate among cor- 
responding plural neurons in input layer 31 and output 
layer 33, neuron Nij in middle layer 32 is judged to have 
the tendency to ignite. Weights of designated neurons 

■ 35 in middle layer 32 and output layer 33 are increased in 
order to obtain such ignition, or b) when the summation 
of an output values in corresponding areas in input layer 
31 and output layer 33, multiplied by predetermined co- 
efficient, is more than a predetermined value, neuron Nij 

40 jn middle layer 32 is judged to have the tendency of ig- 
nition. Weights of designated neurons in middle layer 32 
and output layer 33 are increased in order to obtain such 
ignition. 

The reason why a coefficient is multiplied by the out- 
45 put value of corresponding neuron in output layer 33 is 
to have equal influence on a neuron in middle layer 32 
as an neurons in output layer 33 and input layer 31 . For 
example, when the corresponding neurons in input layer 
31 is 4 and in output layer 33 is 1 , coefficient 4 is multi- 
so plied to the output value in the output layer 33. Conse- 
quently, each neuron in input layer 31 and output layer 
. 33 has equal influence on the ignition distribution of neu- 
rons in middle layer 32. 

The judging method for determining if neurons in 
55 middle layer are to ignite or not is described concretely 
referring Fig. 2 (a), (b) and (c). Fig. 2 (a) shows the ig- 
nition pattern in input layer 31, Fig. 2 (b) shows that in 
middle layer 32, Fig. 2 (c) shows that in output layer 33: 
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in these figures, black circles show ignited neurons, 
white circles show unignited neurons. In this embodi- 
ment, input layer 31 shows the ignition pattern as in the 
diagram according to characteristics data of inputted 
character/letter: the ignition pattern in output layer 33 
have to be what is shown in the diagram so as to rec- 
ognize it. That is, the ignition pattern in input layer 31 in 
the diagram is fixed according to input character/letter: 
the ignition pattern in output layer 33 is what on the com- 
pletion of learning. The ignition pattern in middle layer 

32 is decided according to the ignition pattern of input 
layer 31 and output layer 33. 

The- relationship among neurons in each layer ac- 
cording to Fig. 2A is described below. To each neuron 
in middle layer 32, 4 neurons in input layer 31 and output 
layer 33 are connected in principle. That is to say, the 
neurons connected to neuron Nij in middle layer 32 are 
neurons Nij, N(i+1)j, Ni(i+1), N(i+1)(j+1) in input layer 
31 (the area surrounded by "E2" for neuron N26 in mid- 
dle layer), and M(i-1)(j-1), Nifl-1), N(i-1),j, Nij (the area 
surrounded by "F2" for neuron N26 in middle layer). The 
neurons connected to each neuron N11, N71, N17, and 
N77 on corners in middle layer 32 are the 4 neurons in 
input layer 31 similarly to the above: but those corre- 
sponding to them in output layer 33 are neuron N11, 
N61, N16 and N66 on corners. The neurons connected 
to each neuron Nil, Ni7, Nij, N7j on periphery in middle 
layer 32 are the 4 neurons in input layer 31 similarly to 
the above: but those connected to them in output layer 

33 are neuron N(i-1)1 and Nil for neuron Nil, N(i-1)6 
and Ni6 for neuron Ni7, Ni(j-1) and N1j for neuron Nij, 
and N6(j-1 ) and N6j for neuron N7j. 

Neurons in middle layer 32 are judged to have the 
tendency to ignite when for example, more than 40% 
ignite to plural neurons connected to them in input layer 
31 and output layer 33 in the present embodiment. In 
the example of Fig. 2 (b), 3 neurons ignite among the 
neurons in input layer 31 and output layer 33 connected 
to neuron N26 in middle layer 32 (the 8 neurons in area 
E2 and F2). Therefore, N26 in middle layer 32 is judged 
to be in the tendency to ignite. 2 neurons ignite among 
neurons in input layer 31 and output layer 33 connected 
to neuron N15 in middle layer 32 (6 neurons in area G 
and H): It means that neuron N15 in middle layer 32 has 
not the tendency to ignite. Neurons in middle layer 32 
are judged to have the tendency to show the ignition pat- 
tern in Fig. 2 (b). 

The relationship among neurons in each layer ac- 
cording to Fig. 2B is described below. To each neuron 
in middle layer 32, the area of 4 neurons in input layer 
31 and one neuron in output layer 33 are connected . 
That is to say, the neurons corresponding to neuron Nij 
in middle layer 32 are neurons Nij, N(i+1)j, Ni(i+1), N 
(i+1 )(j+1) in input layer 31 (the area surrounded by "E2" 
for neuron N26 in middle layer), and Nij (neuron N26 for 
neuron N26 in middle layer). In the present embodiment, 
it is settled that the neuron corresponding to neuron Ni7 
on upper edge and neuron N7i on right edge in middle 



layer 32 in Fig. 2 (b) is neuron Ni6 on upper edge and 
neuron N6j on right edge in output layer 33, respectively. 

The neuron in middle layer 32 is judged to have the 
tendency to ignite when the summation of the values in 

s corresponding areas in input layer 31 and output layer 
33, multiplied by coefficient 4, is equal to or more than 
"3", for instance. If the coefficient is "3" and N26 in mid- 
dle layer 32 is judged to have the tendency to ignite. On 
the other hand, as to neuron N33 in middle layer 32, the 

10 total of output value from neuron in corresponding area 
G in input layer 31 is "2", and the output value from cor- 
responding neuron N33 in output layer 33 is "1 ". In this 
case, the value is "6" by summing the output value from 
neuron connected to neuron N33 in middle layer 32 in 

is input layer 31 and the value multiplied the coefficient 4 
to the output value of neuron connected to it in output 
layer 33: neuron N33 in middle layer 32 is judged to be 
■ in the tendency of ignition. The neuron in middle layer 
32 is judged to have the tendency to ignite with the ig- 

20 nition pattern shown in Fig. 2 (b), 

On learning of character/letter recognition in the 
present embodiment, only a predetermined value is in- 
creased in order for the weight of each neuron in middle 
• Iayer32andoutputlayer33toobtaintheignitionpattern 

25 in this way. 

Explaining the increment of weight using Fig. 3, 
when output data O of neuron 1 in middle layer has the 
same value of ignition (for instance "1 "), the weight of 
synapse ("W2" and "W3" in this case) corresponding to 

30 the data inputted from ignited neuron (for instance "12" 
and "13") is, for example, increased 5% among the neu- 
rons in input layer connected to neuron 1 . The synapse 
weight of a neuron in output layer is processed in the 
same way: as described above, the weight of synapse 

35 corresponding to the neuron in middle layer decided to 
be ignited is increased 5%, for example. 

The weight of middle layer 32 and output layer 33 ' 
is increased in order for ignition pattern to be changed 
smoothly among each layer of input layer 31 , middle lay- 

40 er 32 and output layer 33. Here, deciding each neuron 
in input layer 31 and output layer 33 is to ignite on the 
same frequency as much as possible for all input char- 
acters/letters, it is possible for each neuron in middle 
layer 32 to ignite evenly. It is possible to prevent to fall 

45 in local minimum by it, and each neuron in middle layer 
31 ignites evenly in approximation. It means that it is 
possible to avoid the generation of neurons with no ig- 
nition and to make neurons in middle layer 32 work ef- 
ficiently. 

50 The increment of synapse weight on one learning 
changes as in Fig. 4 according to the number of learning. 
The learning of a whole system is executed gradually 
by plural learning, and simultaneously, fine adjustment 
is carried out for fine change. The learning speed is de- 

55 veloped by rapid increase of weight in the beginning. 

In the embodiment above, middle layer 32 is con- 
nected to input layer 31 and output layer 33, that is, the 
number of layers is 3. However, the present invention 
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does not set limits on embodiments as 3 layers of neural 
networks: it is possible to apply it for neural network with 
more than 4 layers. In this case, first of all selecting the 
first middle layer the weight of synapse in the first middle 
layer and output layer is increased such that each neu- 
ron in the middle layer has the tendency of ignition when 
the plural neurons corresponding each other in input lay- 
er and output layer ignite with more than a predeter- 
mined rate. When learning is concluded so far as a pre- 
determined step concerning to all the input characters/ 
letters, adding the second middle layer newly, the weight 
of synapse is increased in the second middle layer and 
the layer connected to output side of it (output layer or 
the first middle layer), considering whether connected 
plural neurons ignite or not with more than a predeter- 
mined rate, in the same way as to the second middle 
layer. Consequently, weight distribution in the case with 

4 layers can be obtained. As to the case with more than 

5 layers, the same process is performed. 

. As to Fig. 2 (a), (b) and (c), respective 4 neurons in 
input layer 31 and output layer 33 are connected to one 
neuron^in middle layer 32. The number of connected 
neurons is not limited by it and can select any number 
according to the necessity. For example, it is possible 
to connect respectively 9 neurons in input layer 31 and 
output layer 33 to one neuron in middle layer 32. 

Here, to neurons N26, N36, N46, N56, N66 and 
N76, areas from E2 to E7 corresponds respectively in 
input layer 31 . In input layer 31 , the neuron in each area 
is included with duplication of adjacent area in right and 
left, and in up and down. Accordingly, the neuron in input 
layer 31 belongs to 4 areas except the edge ones: that 
is, it is connected to 4 neurons in middle layer 32. 

On the other hand, seeing the relationship between 
middle layer 32 and output layer 33, neuron Nij except 
neuron Ni7 and N7i on upper edge and right edge in 
middle layer 32 is connected to 4 neurons in output layer 
33. That is, to neurons N26, N36, N46, N56, N66 and 
N76 in middle layer 32, areas from F2 to F6 in output 
layer 33 correspond. In output layer 33, the neuron in 
each area is included with duplication of adjacent area 
in right and left, and in up and down. Accordingly, the 
neuron in output layer 33 belongs to 4 areas except the 
edge ones and is connected to 4 neurons in middle layer 
32. 

In this way, each neuron in middle layer 32 is con- 
nected to plural neurons in input layer 31 and output lay- 
er 33. Each neuron in input layer 31 and output layer 33 
is connected to plural neurons in middle layer 32. That 
is, the adjacent neuron in input layer 31 and output layer 
33 is also connected to the same neuron in middle layer 
32. Therefore, the adjacent neuron in input layer 31 and 
output layer 33 participates in the judgment of tendency 
to ignite of the same neuron in middle layer 32. 

The input and output data for a neuron is not limited 
to biharized one: it may be multi-value and analog data 
when the data is digital. 

An output layer may be constructed to define a char- 



acter/letter for recognition by all neurons: it is not nec- 
essary to be constructed to express character codes by 
neurons on diagonal line. 

The number of neurons in the layers of 31 , 32 and 
5 33 is not limited to 36: it is possible to settle it according 
to the number of kinds of characters/letters to be recog- 
nized. 

The present invention can be applied to not only 
character/letter recognition but also to pattern recogni- 
io tion and to voice recognition. 

As mentioned above, it is possible to execute effec- 
tive learning on middle layer of neural network by the 
present invention. 

15 

Claims 

1. A learning method for transforming an untrained 
network to a trained network, said network compris- 

20 ing an input layer (31 ),a middle layer (32), and an 
output layer (33), each said layer including a plural- 
ity of neurons, each said neuron being a signal 
processing element, said middle layer being ar- 
ranged between said input and output layers, said 

25 output layer having less neurons than said middle 
layer and said middle layer having less neurons 
than said input layer, wherein each neuron in said 
middle layer is connected to each neuron in said 
input and output layers, said method comprising the 

30 steps of: 

determining an activation pattern of said neu- 
/ rons in said input layer; 

35 determining an activation pattern of said neu- 

rons in said output layer; 

defining a neighborhood of said neurons in said 
input and output layers connected to a neuron 
40 in said middle layer; . * 

determining a rate of activation in said neigh- 
borhood of neurons, said rate being either the, 
total number of neurons activating in said 

45 neighborhood in said input and output layers di- 

vided by the total number of neurons in said 
neighborhood in said input and output layers or 
said rate being defined as the sum of the output 
values of the neighborhood in input layer (31) 

so and the output values of the neighborhood in 

output layer (33) multiplied by a predetermined 
coefficient; 

comparing said rate to a threshold value; 

55 

determining that said neuron in said middle lay- 
er has a tendency to activate if said rate is 
greater than said threshold value; 
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if said neuron in said middle layer has a ten- 
dency to activate, increasing the value of 
weights applied to all neurons in said input layer 
which are connected to said neuron in said mid- 
dle layer and weights applied to all neurons in 
said output layer which are connected to said 
neuron in said middle layer, in order to increase 
the likelihood of activation of said neuron in said 
middle layer; and 

repeating the above steps for each said neuron 
in said middle layer, until a desired activation 
pattern of neurons in said middle layer corre- 
sponding to a final trained network is found. 

• 2. The learning method of claim 1 comprising the 
steps of: 

a. increasing the value of said threshold so that 
less neurons in said middle layer have a ten- 
dency to activate; 

b. comparing said rate in said neighborhood of 
neurons to said increased threshold value; 

25 

c. determining that said neuron in said middle 
layer has a tendency to activate when said rate 
is greater than said increased threshold value; 

4. 

d. if said neuron in said middle layer has a ten- so 
dency to activate, increasing the value of 
weights applied to all neurons in said input layer 
which are connected to said neurons in said 
middle layer and weights applied to all neurons 

in said output layer which are connected to said 35 
neuron in said middle layer, in order to increase 
the likelihood of activation of said neuron in said 
middle layer; and J 

e. repeating steps b.-d. for each neuron in said 40 
middle layer, until a desired activation pattern 

of neurons in said middle layer corresponding 
to a final trained network is found. 

3. The learning method of claim 2 further including the 45 
steps of: 

a. adding an additional middle layer between 
said input layer and said middle layer or be- 
tween said middle layer and said output layer, 50 
said additional middle layer having a plurality 

of neurons; 

b. connecting each neuron in said additional 
middle layer to each neuron in the two layers 55 
adjacent to said additional middle layer; 

c. defining a neighborhood of neurons in said 



adjacent layers connected to a neuron in said 
additional middle layer; 

d. determining a rate of activation in said neigh- 
borhood of neurons, said rate being the total 
number of neurons activating in said neighbor- 
hood divided by the total number of neurons in 
said neighborhood; 

e. comparing said rate to a threshold value; 

f . if said rate is greater than the threshold value, 
increasing the value of weights applied to all 
neurons in one of said adjacent layers which 
are connected to said neuron in said additional 
middle layer and weights applied to all neurons 
in said other adjacent layer which are connect- 
ed to said neuron in said additional middle lay- 
er, in order to increase the likelihood of activa- 
tion of said neuron in said additional middle lay- 
er, and 

g. repeating steps c.-f. for each neuron in said 
additional middle layer, until a desired activa- 
tion pattern of neurons in said additional middle 
layer corresponding to a final trained network 
is found. 

The learning method of claim 3 further comprising 
the steps of: 

a. increasing the value of said threshold so that 
less neurons in said additional middle layer 
have a tendency to activate; 

b. comparing said rate in said neighborhood of 
neurons to said increased threshold value; 

c. determining that said neuron in said addition- 
al middle layer has a tendency to activate when 
said rate is greater than said increased thresh- 
old value; 

d. if said rate is greater than the increased 
threshold value, increasing the value of weights 
applied to all neurons in one of said adjacent 
layers which are connected to said neuron in 
said additional middle layer and weights ap- 
plied to all neurons in said other adjacent layer 
which are connected to said neuron in said ad- 
ditional middle layer, in order to increase the 
likelihood of activation of said neuron in said 
additional middle layer; and 

e. repeating steps b.-d. for each neuron in said 
additional middle layer, until a desired activa- 

. tion pattern of neurons in said additional middle 
layer corresponding to a final trained network 
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is found. 

The learning method of claim 1 , wherein said learn- 
ing method is a method for recognizing characters, 
said activation pattern in said input layer corre- 
sponds to characteristics data of an image, and said 
activation pattern in said output layer corresponds 
to character data of said image. 



Patentanspruche 

1. Lernverfahren zum Uberfuhren eines untrainierten 
Netzwerkes in ein trainiertes Netzwerk, wobei das 
Netzwerk eine Eingangsschicht (31), eine mittlere 
Schicht (32) und eine Ausgangsschicht (33) auf- 
weist und jede Schicht eine Vielzahl von Neuronen 
enthalt, wobei jedes Neuron ein Signalverarbei- 
tungselement darstellt, die mittlere Schicht zwi- 
schen der Eingangsschicht und der Ausgangs- 
schicht angeordnet ist, die Ausgangsschicht weni- 
ger Neuronen als die mittlere Schicht und die mitt- 
lere Schicht weniger Neuronen als die Eingangs- 
schicht aufweist und wobei jedes Neuron in der mitt- 
leren Schicht mit jedem Neuron in der Eingangs- 
schicht und der Ausgangsschicht verbunden ist, 
und wobei das Verfahren folgende Schritte auf- 
weist: 

Bestimmen eines Aktivierungsmusters der 
Neuronen in der Eingangsschicht; 

Bestimmen eines Aktivierungsmusters der 
Neuronen in der Ausgangsschicht; 

Definieren einer Nachbarschaft bezuglich der 
Neuronen in der Eingangs- und Ausgangs- 
schicht, die mit einem Neuron in der mittleren 
Schicht verbunden sind; 

Bestimmen einer Aktivierungsrate in der Nach- 
barschaft der Neuronen, wobei die Rate entwe- 
der die Gesamtanzahl der Neuronen, die in der 
Nachbarschaft in der Eingangs- und Aus- 
gangsschicht aktiviert sind, dividiert durch die 
Gesamtanzahl der Neuronen in der Nachbar- 
schaft in der Eingangs- und Ausgangsschicht 
ist oder die Summe der Ausgangswerte der 
Nachbarschaft in der Eingangsschicht (31 ) und 
der Ausgangswerte der Nachbarschaft in der 
Ausgangsschicht (33) multipliziert mit einem 
vorbestimmten Koeffizienten ist; 



Vergleichen der R 



it einem Schwellwert; 



Bestimmen, daB das Neuron in der mittleren 
Schicht eine Tendenz zum Aktivieren hat, wenn 
die Rate groGer ist als der Schwellwert; 



falls das Neuron in der mittleren Schicht eine 
Tendenz zu aktivieren hat, Erhohen der Ge- 
wichtswerte, die den Neuronen in der Ein- 
gangsschicht zugefugt werden, die mit dem 
Neuron in der mittleren Schicht verbunden sind 
und Erhohen derGewichte, die alien Neuronen 
der Ausgangsschicht zugefugt werden, die mit 
dem Neuron in der mittleren Schicht verbunden 
sind, umdieWahrscheinlichkeitderAktivierung 
des Neurons in der mittleren Schicht zu erho- 
hen; und 

Wiederholen der obigen Schritte fur jedes Neu- 
ron in der mittleren Schicht, bis ein gewunsch- 
tes Aktivierungsmuster von Neuronen in der 
mittleren Schicht entsprechend einem 
schlieBlich trainierten Netzwek gefunden ist. 

Lernverfahren nach Anspruch 1, weiterhin enthal- 
tend die folgenden Schritte: 

a) Erhohen des Schwellwertes, so da3 weniger 
Neuronen in der mittleren Schicht eine Tendenz 
zum Aktivieren haben; 

b) Vergleichen der Rate in der Nachbarschaft 
der Neuronen mit dem erhohten Schwellwert; 

c) Bestimmen, daB das Neuron in der mittleren 
Schicht eine Tendenz zum Aktivieren hat, wenn 
die Rate groRer ist als der erhohte Schwellwert; 

d) wenn das Neuron in der mittleren Schicht ei- 
ne Tendenz zum Aktivieren hat, Erhohen der 
Gewichtswerte, die alien Neuronen in der Ein- 
gangsschicht zugefugt werden, die mit dem 
Neuron in der mittleren Schicht verbunden sind 
und Erhohen der Gewichtswerte der Neuronen 
in der Ausgangsschicht, die mit dem Neuron in 
der mittleren Schicht verbunden sind, urn die 
Wahrscheinlichkeit der Aktivierung des Neu- 
rons in der mittleren Schicht zu erhohen; und 

e) Wiederholen der Schritte b) bis d) fur jedes 
Neuron der mittleren Schicht, bis ein ge- 
wunschtes Aktivierungsmuster von Neuronen 
in der mittleren Schicht entsprechend einem 
trainierten Netzwerk gefunden ist. 

Lernverfahren nach Anspruch 2, weiterhin aufwei- 
send die Schritte: 

a) Addieren einer zusatzlichen mittleren 
Schicht zwischeii der Eingangsschicht und der 
mittleren Schicht oder zwischen der mittleren 
Schicht und der Ausgangsschicht, wobei die 
zusatzliche mittlere Schicht eine Vielzahl von 
Neuronen aufweist; 
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b) Verbinden jedes Neurons der zusatzlichen 
mittleren Schicht mit jedem Neuron in den zwei 
Schichten, die unmittelbar an die zusatzliche 
mittlere Schicht angrenzen; 

c) Definieren einer Nachbarschaft von Neuro- 
nen in den angrenzenden Schichten, die mit ei- 
nem Neuron in der zusatzlichen mittleren 
Schicht verbunden sind; 

d) Bestimmen einer Aktivierungsrate in der 
Nachbarschaft der Neuronen, wobei die Rate 
die Gesamtanzahl der Neuronen, die in der 
Nachbarschaft aktivieren, dividiert durch die 
Gesamtanzahl der Neuronen in der Nachbar- 
schaft ist; 

e) Vergleichen der Rate mit einem Schwellwert; 

f) wenn die Rate groRer ist als der Schwellwert, 
Erhohen der Gewichtswerte fOr alle Neuronen 
in einer der angrenzenden Schichten, die mit 
dem Neuron in der zusatzlichen mittleren 
Schicht verbunden sind und Erhohen der Ge- 
wichte fur alle Neuronen in der anderen an- 
grenzenden Schicht, die mit dem Neuron in der 
zusatzlichen mittleren Schicht verbunden sind, 
urn die Wahrscheinlichkeit der Aktivierung des 
Neurons in der zusatzlichen mittleren Schicht 
zu erhohen, und 

g) Wiederholen der Schritte c) bis f) fur jedes 
Neuron in der zusatzlichen mittleren Schicht, 
bis ein gewunschtes Aktivierungsmuster der 
Neuronen in der zusatzlichen mittleren Schicht 
entsprechend einem trainierten Netzwerk ge- 
funden ist. 

Lernverfahren nach Anspruch 3, weiterhin enthal- 
tend die Schritte: 

a) Erhohen des Schwellwertes, so daf3 weniger 
Neuronen in der zusatzlichen mittleren Schicht 
eine Aktivierungstendenz aufweisen; 

b) Vergleichen der Rate in der Nachbarschaft 
der Neuronen mit dem erhohten Schwellwert; 

c) Bestimmen, daf3 das Neuron in der zusatzli- 
chen mittleren Schicht eine Aktivierungsten- 
denz hat, wenn die Rate groRer ist als der er- 
hohte Schwellwert; 

d) wenn die Rate groBer ist als der erhohte 
Schwellwert, Erhohen der Gewichtswerte, die 
alien Neuronen in einer der angrenzenden 
Schichten zugefugt werden, die mit dem Neu- 
ron in der zusatzlichen mittleren Schicht ver- 



bunden sind und Erhohen der Gewichtswerte 
furalle Neuronen in deranderen angrenzenden 
Schicht, die mit dem Neuron in der zusatzlichen 
mittleren Schicht verbunden sind, urn die Akti- 
vierungswahrscheinlichkeit fur das Neuron in 
der zusatzlichen mittleren Schicht zu erhohen; 
und 

e) Wiederholen der Schritte b) bis d) fur jedes 
Neuron in der zusatzlichen mittleren Schicht, 
bis ein gewunschtes Aktivierungsmuster der 
Neuronen in der zusatzlichen mittleren Schicht 
entsprechend einem trainierten Netzwerk ge- 
funden ist. 

Lernverfahren nach Anspruch 1, wobei das Lern- 
verfahren ein Verfahren zum Erkennen von Zeichen 
ist, und wobei das Aktivierungsmuster in der Ein- 
gangsschicht charakteristischen Daten eines Bil- 
des entspricht und das Aktivierungsmuster in der 
Ausgangsschicht charakteristischen Daten des Bil- 
des entspricht. 



25 Revendications 

1. Procede d'apprentissage permettant de transfor- 
mer un reseau non forme en un reseau forme, ledit 
reseau comprenant une couche d'entree (31), une 
30 couche mediane (32) et une couche de sortie (33), 
chacune desdites couches comprenant une plura- 
lity de neurones, chacun desdits neurones etant un 
element de traitement de signaux, ladite couche 
mediane etant placee entre ladite couche d'entree 
35 et ladite couche de sortie, ladite couche de sortie 
ayant moins de neurones que ladite couche media- 
ne et ladite couche mediane ayant moins de neu- 
rones que ladite couche d'entree, dans lequel cha- 
que neurone de ladite couche mediane est relie a 
40 chaque neurone de ladite couche d'entree et de la- 
dite couche de sortie, ledit procede comprenant les 
etapes de : 

. determination d'une configuration d'activation 
desdits neurones dans ladite couche d'entree ; 
determination d'une configuration d'activation 
desdits neurones dans ladite couche de sortie ; 
definition d'un voisinage desdits neurones 
dans ladite couche d'entree et dans ladite cou- 
che de sortie, relies a un neurone dans ladite 
couche mediane ; 

determination d'un taux d'activation dans ledit 
voisinage des neurones, ledit taux etant soit le 
nombre total des neurones entrant en fonction 
dans ledit voisinage dans ladite couche d'en- 
tree et dans ladite couche de sortie, divise par 
le nombre total de neurones dans ledit voisina- 
ge dans ladite couche d'entree et dans ladite 
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couche de sortie, soil defini comme la somme 
des valeurs de sortie du voisinage dans la cou- 
che d'entree (31) et des valeurs de sortie du 
voisinage dans la couche de sortie (33), multi- 
pliee par un coefficient predetermine ; 
comparaison dudit taux et d'une valeur de 
seuil; 

determination du fait que ledit neurone dans la- 
dite couche mediane a tendance a entrer en 
fonction si ledit taux est superieur a ladite va- 
leur de seuil ; 

si ledit neurone dans ladite couche mediane a 
tendance a entrer en fonction, augmentation de 
la valeur des poids appliques a tous les neuro- 
nes de ladite couche d'entree, qui sont relies 
audit neurone de ladite couche mediane, et de 
la valeur des poids appliques a tous les neuro- 
nes de ladite couche de sortie, qui sont relies 
audit neurone de ladite couche mediane, afin 
d'augmenter la probabilite d'activation dudit ■ 
neurone de ladite couche mediane ; et 
repetition des etapes ci-dessus pour chacun 
desdits neurones dans ladite couche mediane, 
jusqu'a ce que soit trouvee une configuration 
d'activation desiree des neurones dans ladite ■ 
couche mediane, correspondant a un r^seau fi- 
nal forme, 

Procede d'apprentissage selon la revendication 1 , 
comprenant les etapes de : 

a. augmentation de la valeur dudit seuil afin que 
moins de neurones de ladite couche mediane 
aient tendance a entrer en fonction ; 

b. comparaison dudit taux dans ledit voisinage 
des neurones et de ladite valeur de seuil 
augmentee ; 

c. determination du fait que ledit neurone dans 
ladite couche mediane a tendance a entrer en 
fonction lorsque ledit taux est superieur a ladite 
valeur de seuil augmentee ; 

d. si ledit neurone dans ladite couche mediane 
a tendance a entrer en fonction, augmentation 
de la valeur des poids appliques a tous les neu- 
rones de ladite couche d'entree, qui sont relies 
audit neurone de ladite couche mediane, et de 
la valeur des poids appliques a tous les neuro- 
nes de ladite couche de sortie, qui sont relies 
audit neurone de ladite couche mediane, afin 
d'augmenter la probabilite d'activation dudit 
neurone de ladite couche mediane ; et 

e. repetition des etapes b. a d. pour chaque 
neurone dans ladite couche mediane, jusqu'a 
ce que soit trouvee une configuration d'activa- 
tion desiree des neurones dans ladite couche 
mediane, correspondant a un reseau final for- 
me. 



3. Procede d'apprentissage selon la revendication 2, 
comprenant en outre les etapes de : 

a. addition d'une couche mediane supplemen- 
taire entre ladite couche d'entree et ladite cou- 
che mediane, ou entre ladite couche mediane 
et ladite couche de sortie, ladite couche media- 
ne supplementaire ayant une pluralite de 
neurones; 

b. connexion de chaque neurone de ladite cou- 
che mediane supplementaire a chaque neuro- 
ne des deux couches adjacentes a ladite cou- 

, che mediane supplementaire ; 

c. definition d'un voisinage de neurones dans 
lesdites couches adjacentes, relies a un neuro- 
ne de ladite couche mediane supplementaire ; 

d. determination d'un taux d'activation dans le- 
dit voisinage de neurones, ledit taux etant le 
nombre total de neurones entrant en fonction 
dans ledit voisinage, divise par le nombre total 
de neurones dans ledit voisinage ; 

e. comparaison dudit taux et d'une valeur de 
seuil ; 

f. si ledit taux est superieur a la valeur de seuil, 
augmentation de la valeur des poids appliques 
a tous les neurones dans I'une desdites cou- 
ches adjacentes, qui sont relies audit neurone 
de ladite couche mediane supplementaire; et 
de la valeur des poids appliques a tous les neu- 
rones, dans I'autre desdites couches adjacen- 
tes, qui sont relics audit neurone de ladite cou- 
che mediane supplementaire, afin d'augmenter 
la probabilite d'activation dudit neurone de la- 
dite couche mediane supplementaire, et 

g. repetition des etapes c. a f. pour chaque neu- 
rone de ladite couche mediane supplementai- 
re, jusqu'a ce que soit trouvee une configura- 
tion d'activation desiree des neurones dans la- 
dite couche mediane supplementaire, corres- 
pondant a un reseau final forme. 

4. Procede d'apprentissage selon la revendication 3, 
comprenant en outre les etapes de : 

augmentation de la valeur dudit seuil afin que 
moins de neurones de ladite couche mediane 
supplementaire aient tendance a entrer en 
fonction ; 

b. comparaison dudit taux dans ledit voisinage 
des. neurones et de ladite valeur de seuil 
augmentee ; 

c. determination du fait que ledit neurone dans 
ladite couche mediane a tendance a entrer en 
fonction lorsque ledit taux est superieur a ladite 
valeur de seuil augmentee ; 

d. si ledit taux est superieur a la valeur de seuil, 
. augmentation de la valeur des poids appliques 

a tous les neurones dans I'une desdites cou- 
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ches adjacentes, qui sont relies audit neurone 
de ladite couche mediane supplementaire, et 
de la valeur des poids appliques a tous les neu- 
rones, dans I'autre desdites couches adjacen- 
tes, qui sont relies audit neurone de ladite cou- 5 
che mediane supplementaire, afin d'augmenter 
la probability d'activation dudit neurone de la- 
dite couche mediane supplementaire ; et 
e. repetition des etapes b. a d. pour chaque 
neurone de ladite couche mediane supplemen- 10 
taire, jusqu'a ce que soit trouvee une configu- 
ration d'activation desiree des neurones dans 
ladite couche mediane supplementaire, corres- 
pondent a un reseau final forme. 

15 

Precede d'apprentissage selon la revendication 1, 
dans lequel ledit precede d'apprentissage est un 
procede de reconnaissance des caracteres, ladite 
configuration d'activation dans ladite couche d'en- 
tree correspondent a des donnees de caracteristi- 20 
ques d'une image, et ladite configuration d'activa- 
tion dans ladite couche de sortie correspondent a 
des donnees de caracteres de ladite image. 
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